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Abstract: 

University education is characterized by the atmosphere and opportunities offered to 
students to help master new skills, develop existing ones, realize and accept cultural 
diversity, hence broaden their horizons. Compared to past, students now have 
numerous alternatives either in the university in their home country or abroad. 
According to the statistics, 10% of the students graduated from university in the mid- 
1990s had studied abroad for different time periods (Jahr, Schomburg & Teichler, 2002). 
Undoubtedly, this number has probably increased given the fact that developments in 
technology have resulted in obvious easiness in communication and transportation. 
More importantly, there are now student exchange programs like Erasmus+ that help 


support student mobility. 
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1. Introduction 


Launched in 1987, Erasmus+ is an EU’s program which aims to ‘contribute to the 
Europe 2020 Strategy for growth, jobs, social equity and inclusion, and to the aims of 
ET2020, the EU’s _ strategic framework for education § and _ training’ 
(http://ec.europa.eu/programmes/erasmus-plus/node en). The statistics show that in 
2015, 678,000 people studied, trained or volunteered abroad and there were 69,000 


organizations involved in Erasmus+. Through Erasmust+, students have the opportunity 
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to pursue their studies in another country in Europe for at least a period of six months, 
which might be extended to one year, and even do their internship for three months 
before they graduate from university. Students taking this opportunity are also given a 
certain amount of grant for their expenses. Research shows that many students like to 
go abroad through Erasmus+ because of the opportunities to learn about a different 
culture, benefit from cheap and easy travel overseas and make a change in their CV for 
their future career (Tekin & Gencer, 2013). 

Turkey is among the countries where universities and university students benefit 
from Erasmust+. There are 184 universities in Turkey and as of 2011, 131 of them benefit 
from Erasmus+ student exchange program. The demand for the program is high, yet it 
is unlikely to send all the applicants to Europe taking the amount of grant spared for 
student mobility and the possible quotas agreed by the host universities. This makes the 
application and admission process a competitive and challenging one with certain 
requirements, and only the ones meeting them are eligible for the program. These 
requirements include a minimum GPA of 2,20 and a proof of good language proficiency 
level. 

This competitive process brings language assessment as one of the issues to be 
considered. As language proficiency is essential to survive and pursue their studies 
abroad, students take language proficiency tests measuring their overall mastery of 
English language abilities. Taken the diversity of students in terms of their majors and 
language background, developing and deploying language tests is a challenge itself 
because studying abroad requires mastery in both productive, i.e. speaking and writing 
and receptive, i.e. reading and listening, language skills now that students merge into a 
new community where the medium of communication is English, if not only. This 
suggests that students should be competent in oral and written skills with correct 
register in both social and academic contexts. 

A quick research on the Internet reveals that universities having Erasmus+ 
student exchange program in Turkey deploy language proficiency tests differing in 
content and construct. Given the fact that these tests play an important role in making 
decisions by universities, how language proficiency is tested is of crucial importance. 
As McNamara (2000) stated, proficiency tests do not aim to assess students’ success 
considering the language instruction they have received previously, but look to the 
future situation of language use. Some universities have candidate students sit a 
multiple-choice test, in which grammar, vocabulary, reading and translation skill of 
students are tested, with the inclusion of listening in some other universities. There are 
some examples of proficiency tests with a writing component, in which candidate 


students are required to write an essay. There are few incidents of a speaking test given 
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to the students. Also, the number of questions in the multiple-choice test varies as well 
as the item types used. As Erasmus+ is a Europe-based program, the language 
proficiency of students is expressed according to the Common European Framework of 
Reference for Languages (CEFR), from A1 (the lowest) to C2 (the highest). The language 
proficiency tests in Turkey also range from A2 to Cl in the language level they are 
developed. 

All assessment tools are meant to be valid, reliable and practical (Madsen, 1983; 
McNamara, 1996). As for the language proficiency tests for student mobility, namely 
Erasmus+ student exchange program, they seem to heavily rely on multiple-choice 
discrete item tests, which might stem from the practicality issues due to high number of 
applicants and low number of raters to test productive skills. In other words, there 
seems to be that practicality issue plays a major role in decision making when it comes 
to testing language proficiency, contrary to the fact that students need to be competent 
in all language skills to survive in both academic and social contexts abroad as 


mentioned earlier. 
2. Aim of the Study 


This present study aims to determine whether it is possible to substitute scores 
from different dimensions of English language skills. This study particularly seeks 
answers to the following research questions: 

1. Is there a significant correlation between student scores from receptive skills (i.e. 
Booklet — multiple choice) and productive skills (i.e. written and oral tests — open 
response)? 

2. Does the correlation between receptive and productive skills tests make it 
possible to substitute one another? 

3. What are the perceptions of test takers about what should constitute a language 


proficiency test for student mobility? 
3. Setting 


Anadolu University, founded in 1982 having its roots back in 1958, is a state 
university in Turkey. Anadolu University is among the largest universities in Turkey 
with its modern, dynamic and innovative nature, housing 17 faculties, 3 applied 
schools, 4 vocational schools and 9 graduate schools located on two different campuses. 
As of 2016, the total number of students is 1,140,288, of which 4,089 are foreign 
students. 
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AUSFL hosts undergraduate students in their first year at university and 
provides them a one-year intensive language course because the medium of instruction 
in some faculties is English, French, German or Russian, thus they need to be proficient 
in the required language to pursue their courses in their undergraduate studies. For all 
the languages mentioned before, a language proficiency test is deployed for exemption 


from this one-year education. 


4. The English Language Proficiency Test 


The ELPT for Erasmus+ student exchange program is administrated by AUSEL. 
The test consists of three components, namely Booklet, Written and Oral Tests. In the 
Test, the items in all components are aligned with the learning outcomes in the Global 
Scale of English (GSE). The GSE is a granular academic scale enriching the CEFR and 
showing student level and progress in fine progression (http://www.english.com/gse). 


Table 1 gives a quick overview of the test. 


Table 1: ELPT at a Glance 


Test # Item Language Skills / Duration Weight 
Component | Items Type(s) Systems (%) 


Discrete item Listening . 
: : 75 mins 
Multiple Choice Grammar veaas 
Booklet een after the listening 
(Fill-in blanks, cloze tests, Vocabulary j 
; ; section 
sentence completion, etc) Reading 


; Open response 
Written : 
Test 2 Two alternatives Writing 60 mins 20% 
es 
(one to be written) 

Open response ; 

Oral ; epee : 15 mins 
3 (2 items done individually, Speaking ; 

Test ; j j for each pair 

1 item in pairs) 


As seen in Table 2, Booklet is the multiple-choice component of the Test, aiming to 


measure students’ language abilities in two receptive skills and two language systems. 
It is administrated in one session with the invigilation of English language teachers 
working at AUSFL. The Written test aims at measuring students’ abilities in expressing 
themselves in written English and student papers are separately graded by two 
different raters using criteria for task achievement, organization, grammatical and 
lexical competence. In the Oral Test, two different raters grade student performances 
just after the students leave the test room, where all the test process is video recorded. 


As for the weight of the components in the Test, Written and Oral components have 
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20% weight each, and the Booklet has % 60, which makes the overall test score showing 
the proficiency level of test takers. In this study, the Test deployed in 2016-2017 


Academic Year was analyzed. 
5. Participants 


The number of participants across their majors taking the 2016-2017 Test 


participants is given in Table 2 . 


Table 2: The Participants 


School / Faculty / Institution # of test takers 
Faculty of Pharmacy 12 
Faculty of Humanities 15 
Faculty of Education * 25 
Faculty of Science * 8 
Faculty of Aeronautics and Astronautics 11 
School of Music and Drama 14 
Faculty of Law 222 
Faculty of Economics and Administrative Sciences * 11 
Faculty of Communication Sciences * 43 
Faculty of Architecture and Design 55 
Faculty of Engineering * 53 
Faculty of Health Sciences 

Faculty of Sport Sciences 6 
Graduate School of Fine Arts 4 
Graduate School or Sciences 26 
Graduate School of Social Sciences 19 
Graduate School of Earth and Space Sciences 1 
Total 421 


* The medium of instruction is fully or partially English. 


As seen in Table 2, a total of 421 participants took the Test in the Fall Term of 2016-2017 
Academic Year. The participants vary in their major, ranging from School of Music and 
Drama where the medium of instruction is Turkish to Faculty of Engineering where the 
instruction is done in English. The majority of the participants are undergraduate 
students (88,2%) and the minority is graduate students (11,8%). 
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6. Data Collection and Analysis 


In this study, the scores of students taking the ELTP, administered by AUSFL in 
three different sessions, for Erasmus+ Student Exchange Program to study abroad are 
used. In the first part of the study, the relationship between the scores of 421 students 
from the Booklet, Written and Oral tests was analyzed. In the second part, the Pearson 
correlation coefficient between the students’ overall scores from the Written and Oral 
tests and the scores from the Booklet was calculated. The Pearson correlation coefficient 
gives a value ranging in value from -1 to +1. This value indicates whether a linear 
relationship between two continuous variables exists. In other words, the larger the 
absolute value of the Pearson coefficient, the stronger the relationship between the 
variables. If this value is negative, it is deduced that the relationship between the 
variables is reversed and if it is positive, it is in the same direction. Values close to 0 or 0 
mean that there is no relationship between variables. As absolute value, values between 
0 and 30 are low; values between .31 and .70 are moderate, and values between .71 and 
1.00 mean a high correlation between variables (Biiyuk6oztiirk, 2009). Finally, 
considering the values of the Pearson correlation coefficient, the possibility of 
substituting the tests administered in different sessions to determine English 
proficiency has been discussed. 

In the last part of the study, students’ Written Test outputs are used to explore 
their opinions about what should constitute a language proficiency test for student 
mobility in their response to the following question: “What language skill(s) should be 
tested in Erasmus Exams?” 

Nearly half of the students responded this question and 80 of the students who 


scored 80 and above out of 100 were analyzed to identify the recurring themes. 
7. Results 


In this study, the correlation between the participants’ scores on the Booklet, 
Written and Oral Tests (Research Question 1); and the Pearson correlation coefficient 
between the students’ overall scores from the Written and Oral tests and the scores 
from the Booklet (Research Question 2) were calculated. 


As for research question 1, the results are shown in Table 3 below. 
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Table 3: The correlation between scores from the Booklet and the Written & Oral Tests 


Pearson Correlation Coefficient 
Component Booklet Written Oral 
Booklet - A5* 41* 
Written 45* - 32” 
Oral A1* .32* - 
*p<.01 


As seen in Table 3, the correlation between the three components of the ELPT was 
positive and statistically significant. It is seen that the values change between .32 and 
.45. The highest correlation score, .45, is between the scores taken from the Written Test 
and the Booklet, the multiple-choice component of ELPT. The correlation value between 
students’ scores on the Booklet and Oral Test was calculated as .41. In addition, the 
correlation between the Written and Oral Tests, measuring productive skills, was 
calculated .32. The values indicate a moderate correlation between the variables. 


As for research question 2, the findings are shown Table 4. 


Table 4: The correlation coefficient between students’ scores from the Booklet and 
their overall scores from the Written and Oral Tests 


Pearson Correlation Coefficient 


Component Written & Oral 
Booklet 1 
*p<.0l 


As seen in Table 4, there is a statistically significant correlation between the students’ 
scores from the Booklet, the multiple-choice components of the ELPT, and their overall 
scores from the Written and Oral Tests measuring productive skills. The correlation 
value of .51 indicates a moderate relation between the variables. 

As for research question 3, almost half of the students stated listening and 
speaking should definitely be tested to communicate effectively with others; to survive 
in a foreign country; to follow courses at university by stating the fact that listening and 
speaking work collaboratively, hence inseparable and indispensible. The following 
extracts are takes from students’ outputs: 

“... An exchange student should be able to comprehend what others are saying, 


especially since their education will continue abroad. ... ” 
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“I can’t imagine that a student wants to go abroad and cannot speak the language..” 


“ ... When we talk about daily tasks, the first thing comes up to our minds is speaking 


because human beings cannot survive without speaking. .. “ 


“... When you arrive in the city, you must contact someone to solve your problem. Also, 
you need to understand what they are saying. If you are asking something, you need to 


understand the answer. ...” 


Nearly half of the students emphasized the importance of reading and writing 
skills for Erasmus+ students in order to follow the courses at university because they 
are required to read to study the content and write assignments. It would also help 
them discover new things regarding life in general and their majors, which would be 


achieved by applying effective reading strategies to whatever they read. 


“... We'll go to a foreign country for education. We'll read, write and have exams at 


‘a 


school, so we have to be good in them. ...’ 


“ ... Erasmus students will take many exams in their schools. If they don’t have enough 
writing skills, they will probably get low grades and a bad Erasmus experience. ...” 


“. Reading and writing are incredibly important for academic life. Most of [the] time 
you have to read essays and write essays. Reading helps you to write brilliant essays. 


While you are reading, you see how to use grammar and learn more words. ... “ 


“... Students’ reading skills must be at the top of the range because all lesson materials 
are in the language used in that country, so if your reading skill is not at [a] good level, 


Mh 


education will be harder for you. ... 


Whatever skills are prioritized in students’ outputs, nearly all of them 
emphasized the importance of having a good command of grammar and vocabulary 
knowledge because they would provide a basis for all language skills preventing 
distortions in meaning; increasing comprehension as well as accuracy and fluency. The 


following extracts are taken from students’ outputs: 


“. The mastery of all the skills means a higher chance of achieving the goals of an 


exchange program like Erasmus+ ...” 
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“... These skills are really dependent on each other. ... “ 
“... it’s fair to say that all language skills are required to achieve the program’s goals. ...” 


“. If we want to write correct sentences, we have to learn grammar rules very well. ... 
Vocabulary.. knowing a lot of words helps us to be better in other skills. If we don’t learn 


a lot of words, we don’t understand people, we don’t write and read. ... “ 
8. Discussion 


Tests designed to measure foreign language proficiency are intended to reveal 
test takers’ abilities in different language skills. Tests with multiple choice items are 
usually developed to measure language abilities in receptive skills, i.e. listening and 
reading, and language systems or in other words sub-skills, i.e. grammar and 
vocabulary. On the other hand, written and oral tests are designed to evaluate test 
takers’ abilities in productive skills. 

In this study, the correlation between the scores of test takers from three different 
components of the ELPT administrated by AUSFL, a language proficiency test to 
determine the candidates for Erasmus+ Student Exchange Program to study abroad was 
calculated. The aim was to determine whether it was possible to deploy one or two 
components of the test, rather than the whole, and use their overall score to measure 
language proficiency, rather than having students take multiple tests for different skills 
in different sessions. In the literature, there are studies stating that if found a positive 
and statistically significant relationship between tests measuring different dimension of 
the same psychological structure, then the most practical one(s) may be deployed only. 
Practicality is likely to affect decisions because resources required to deploy tests may 
be limited (Bachman & Palmer, 1996). For example, item writers or test developers as 
human resources; space, equipment and materials as material resources; and time for 
designing, writing, administrating, scoring and the like as time resources. This might be 
the reason why productive skills are ignored on language proficiency tests given the 
fact that there is always a high demand for Erasmus+ student exchange program. 

The preliminary aim of this study was to find out the correlation value between 
the Booklet, the Written and the Oral Tests. The results revealed that there was a 
positive and statistically significant correlation. It is important to note that the 
correlation value, .30, is close to the lower limit for the moderate level correlation value. 
This finding suggests that student scores from any of these components may not fully 


or accurately represent their language abilities in the other language skills. In other 
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words, deploying only one of these tests may not prove effective in measuring students’ 
abilities in other skills which are not tested. 

Another aim of the study was to find out the correlation between student scores 
from the Booklet and their overall scores from the Written and Oral Test scores. The 
correlation value was found .51, which was relatively higher than the previously 
mentioned correlation values between the components of the ELPT. This correlation 
value indicates a positive and moderate relationship between the two. Considering this, 
it might be concluded that scores from the Booklet and the overall scores of the Written 
and Oral Tests may not substitute each other. In other words, a score from the Booklet 
may not be a representative of their abilities in the Written and Oral tests aiming to 
measure test takers’ language abilities in different skills. 

Consequently, considering the results of the study, it might be concluded that 
there was a moderately statistically significant correlation between the Booklet, the 
Written and the Oral tests. However, the fact that these relations are not at a very high 
level does not make it possible to substitute these three tests for each other, thus 
reducing the number of sessions, hence the tests, administered. Correlation between 
different dimensions of the same skill is theoretically expected. However, in this study 
the correlation values are not very high. This might be linked to the fact that English 
language is a whole and all language skills are integrated, hence inseparable from and 
highly indispensible for each other, operating in harmony. In other words, each 
component and psychological sub-dimension constitutes the whole, namely language 
proficiency. By taking these findings into consideration, it might be concluded that 
administering multiple-choice tests only and ignoring productive skills due to 
practicality concerns may not prove effective. Written and oral abilities of students 
should also be tested to determine the language proficiency of students, especially the 
ones aiming to pursue their career abroad. This conclusion is also supported by 
students’ perceptions of what should constitute a language test for language 
proficiency. It is important to test productive skills as acknowledged by students 
because language skills and systems are integrated and they all serve the same purpose: 


effective communication. 
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